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1. INTRODUCTION 

Artificial intelligence (AI) is one of the newest fields in science and engineering. The Turing test is 
used to determine if a system is intelligent or not. A machine passes the test if a human inquisitor can’t 
identify if the written responses are from a machine or a person after posing a series of written questions. 

Natural language processing (NLP) is an AI application that represents a computer science 
discipline that converts human voice into commands and is classified under human-computer interaction 
category. The majority of NLP algorithms are built using machine learning (ML). NLP is employed here to 
allow the user to interact with household appliances using a human voice based on machine learning [1]-[7]. 

Recently, what is known as the internet of things (IoT) has made it possible to connect machines to 
machines. Building new services over the network using radio-frequency identification (RFID) technology 
combined with sensors and/or mobile applications helps a lot in positively driving IoT applications [8]-[16]. 
Intelligent home systems have grown rapidly in recent years and they are being at the core of IoT, 
information technology, and communication technologies. In a smart home system, power and lighting 
consumption can be controlled and monitored remotely using web interface or mobile devices. To apply IoT, 
Arduino and Raspberry Pi have been used to translate the commands derived from NLP into actions. 
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This paper integrates IoT and voice recognition technology in addition to NLP-based AI to improve 
smart homes systems control. The paper is divided into five sections; section 1 introduces the study; section 2 
highlights relevant works; section 3 details the proposed system and its components; section 4 displays 
system implementation; and section 5 concludes the study. IoT and AI are now integral components of 
modern systems, and doping IoT with AI has been documented in numerous studies. 

Dey et al. [17] used only the wireless sensors to build home automation systems, using the hypertext 
markup language (HTML) web page as interface to correlate between Arduino boards and web page. 
David et al. [18] presented an alternative method that uses a Raspberry Pi board rather than an Arduino 
board. They presented Android applications that might be used to control the appliances’ functions, however 
voice commands are not supported. Kumar and Shimi [19] used an Arduino board, a relay circuit, and a voice 
recognition module in addition to an adjustable bed to create a home automation system with voice 
commands, however the functionality was limited. Satria et al. [20], facilitates mobile device Bluetooth as an 
interface to govern the home appliances, but the main drawback in this work was the Bluetooth coverage 
distance limitation. Sen et al. [21], overcame this drawback by switching from Bluetooth to the global system 
for mobile (GSM) for home appliance control, resulting in a progressive increase in the total cost of the home 
automation system deployment. Erić et al. [22], demonstrated the differences between text to speech and 
speech to voice engines in the context of a smart home system. Baby et al. [23], proposed creating a smart 
home system using a chatbot based on natural language processing. A web application has been used to 
facilitate user authentication for security reasons. Quadri and Sathish [24], developed an IoT monitoring system 
that allows them to control the door wirelessly using a smartphone and a Raspberry Pi. Yousef and Torad [25] 
created a well-organized conversational AI agent that can understand and analyze human behavior based on 
day-to-day encounters and interactions, in addition to the State of the art of voice control for home/domotic 
systems mentioned in [26]. 

Smart home systems-related studies have been evaluated, summarised, and contrasted in Table | for 
the sake of brevity. The report also provided a comparison of the proposed smart home system to highlight its 
key characteristics in contrast to existing systems. The suggested system, as shown in the table, seeks to 
overcome the constraints of the existing system and is accompanied with a real-world implementation 
scenario for validation reasons. The contributions of this paper, as well as the model presented in [25] will be 
integrated with IoT model to manipulate a more efficient smart home system. 


Table 1. Summary of smart home systems 


Focus criteria 


F 5 Q i = Tz 
Ze on %2 a E pg aa Z B & zg &B 
G gfe n E pE 
27 gg & g E GR fe J Z EPRE 
© a < a or i © a 
Dey et al. [17] V y V WiFi Arduino Mega y 
David et al. [18] y y y Bluetooth/WiFi Arduino Mega V vV 
Kumar and V Arduino Mega y 
Shimi [19] 
Satria et al. [20] y y Bluetooth Arduino Mega y 
Sen et al. [21] V y GSM Arduino Mega V 
Erić et al. [22] y V WiFi Arduino V vV 
Baby et al. [23] V y V y WiFi Raspberry Pi y V 
Quadri and y V V y WiFi Raspberry Pi Vv 
Sathish [24] 
Proposed system y V y WiFi Arduino Mega + Raspberry Pi V V Vv V 
+ node MCU 


2. RESEARCH METHOD 

The proposed system architecture is depicted in Figure 1. This work aims to make voice commands 
in human natural language a better and easier way to connect with smart home appliances or web 
applications. Also a recommendation system to control home appliances is provided. A back-end server is 
configured, especially for heavy processing operations such as NLP, recommendation system, and image 
processing. The server is created via an open source microframework for python (flask). The present system 
includes the following main modules: 
a) Authentication module 
b) Remote voice control module 
c) Smart recommendation module 
d) Weather information module 
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Figure 1. The proposed system architecture 


2.1. Authentication module 

The proposed system can perform autontication by two ways. First by captures the user’s face image 
and compares it to the one saved on the system server, and if the two match, the user is authenticated. 
Secondly, a username and password can be used for authentication. It should be noted that the image 
identification is carried out utilizing image processing algorithms via computer vision module using OpenCV 
library. This process can be done through android or web applications. This authentication system is a one-time 
process that increases the credibility of the system. 


2.2. Remote voice control module 

At chat bot, the user’s voice commands are converted from speech to text and then delivered to the 
server via a hypertext transfer protocol (HTTP) request. NLP consists of eight layers as [25]: 

a) Layer 1 (expanding contractions): this layer uses the pros of the regular expressions module supplied in 
the python language. 

b) Layer 2 (tokenization): it is a pre-processing phase achieved to split the incoming string into a list of 
tokens or words. Process done through the natural language toolkit (NLTK) library. 

c) Layer 3 (spelling correction): where the spelling correction algorithm is applied. 

d) Layer 4 (intent and tense detection): text-classifier used for the purpose of detecting user’s intent. 
Naïve-Bayes classifier algorithm was used [17]. 

e) Layer 5 (part-of-speech tagging): tag each of the resulted list of tokens into its suitable part-of-speech 
(POS) tag i.e. proper noun/determiner/preposition. this layer process done through the NLTK library. 

f) Layer 6 (information extraction layer): which facilities the pros resulted from the chunking in addition 
chinking methods supported by the NLTK library for the purpose of extracting the vital chunks from 
user’s input list and aggregating them into a specific list, e.g. [‘coffee machine’, ‘on’]. 

g) Layer 7 (organization): the resulted extracted information list is organized into an exact dictionary 
which contains an appropriate label to every entry, e.g. {‘appliance’: ‘light’, ‘state’: ‘on’}. 

h) Layer 8 (execution): it provides a controlling machine upon all the previous layers. 

The NLP algorithms perform a process on the given texts extracted from the human voice to 
determine the user intent. Accordingly, the needed algorithms are run for realizing this intent. All the 
processes are completely processed and provide the appropriate response which is done by the NLP in the 
order of milliseconds. The system server is connected to Raspberry Pi via message queue telemetry transport 
(MQTT) connectivity protocol. Representational state transfer application program interface (RESTful API)’s is 
utilized to make it more efficient and easier to establish a connection between the server and web or mobile 
applications. 


2.3. Smart recommendation module 

Recommender systems have grown in popularity in recent years, and they are now used in a number 
of applications such as news, movies, books, music, search queries, research articles, social tags, and other 
similar applications. The NLP represents the first step in the recommendation system and is usually followed 
by a ML step. A recommendation system is a subclass of information filtering system that looks for ways to 
predict the preference or rating of the user’s behaviour in relation to various home appliances. As illustrated 
in Figure 2, there are three possible approaches to developing a recommendation system. The approach under 
consideration is a “content-based filtering technique” which collects and analyzes data based on users’ 
behaviour. This is done by using nearest neighbour algorithm, which is characterized by fast computation of 
nearest neighbours. This is still an active zone of research in ML recommender systems. The nearest 
neighbour search implementation involves the brute-force computation of distances between all pairs of 
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points in the dataset. The recommender system considered in this paper is developed in python language 
using three modules, namely sci-kit learn, pandas and numpy. The three modules are referred to in python as 
indicated in Figure 3. 

The recommender class is initially implemented with the aid of its constructor, then the selected 
dataset and user values are continuously passed to the selected pandas module. After that, the chosen data is 
fitted to the proposed algorithm, and a large number of neighbors is determined. Accordingly, the suitable 
behaviour is recommended by the system. 

If a user requests to watch a movie on the Internet, for example, there are a lot of options, creating a 
problem for many Internet users. Therefore, certain delivered information must be filtered, preferred, and/or 
excelled. The recommender system will generate a list of recommended movies based on the user’s previous 
choices. This is accomplished by exploring a vast database of information and delivering personalized 
content to users. 


2.4. Back-end controller module 

Back-end controller is important because it is responsible for the communication between the 
module implemented and all the interfaces, all interfaces discussed in details into next section. The python 
programming language was chosen since its libraries are open source and well-suited to this proposed work. 
The “flask”, a micro web framework developed in python editor, was chosen. The server in question is 
configured using “flask”, and is connected with the android and web interfaces by through a “restful 
application program interface” (API), which uses HTTP requests to get, put, post and delete data. 

For the hardware branch, we use MQTT cloud service. MQTT is a machine-to-machine 
(M2M)/connectivity protocol. It was designed as a very lightweight subscribe/publish messaging transport. 
It is valuable for connections with remote locations where a minor code footprint is essential and/or network 
bandwidth is at top. 


Recommender 
system 


Contented based 
Filtering Technique 


Collaborative Hybrid Filtering 
Filtering Technique Technique 


Model based Memory based 
Filtering Technique Filtering Technique 


Figure 2. Classification of the recommender systems 


Clustering Technique 
Association Technique 
Bayesian Technique 
Neural Technique 


3. RESULTS AND DISCUSSIONS 

The following are presented as proofs of concept: a) movie recommender: it’s used to recommend 
movies for user. This recommendation based on the user dataset. So, based on user behaviours 
recommendation generated for the user; and b) temperature degree: returns the temperature of a specific city 
by means of weather API. (The above two functions are shown in Figure 4). Figure 5 depicts the 
implementation of the main API, which uses the POST (HTTP request) method to receive data from the 
interface, and the NLP module to interpret the user’s intents via chatbot or voice commands. If there is a 
request for activation/ deactivation a certain appliance, the API delivers it to the hardware, which generates a 
response using the mouth module and sends it back to the interface. 
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# Importing the modules 
import numpy as np 
import pandas as pd 
import sklearn 


from sklearn.neighbors import NearestNeighbors 


Figure 3. Importing three modules in python used in ML 


3.1. Air-conditioning module 

The air-conditioner model is implemented using numpy, pandas and scikit-learn python modules. 
To generate a dataset of the user’s behaviour and train on it to generate and predict results based on day and 
time attributes, the proposed classification system processes are performed on the server, and certain data is 
required from the server, which is gathered by the sensors. and make a classification based on the user’s 
previous decisions and user’s data to classify turning the air-conditioner on or off then send the value that has 
been predicted to the server. Decision tree is the classification model used here. Such data includes: 
a) Date: the date on which the decision has been made. 
b) Time: the time in which the decision has been made. 
c) Interior value: temperature degree inside. 
d) Exterior value: temperature degree outside. 
e) User value: the user decision to turn the conditioner on or off. 


def recmmend(mgenra): 
Movies = pd.read_csv( 'Core/DataSets/movie_metadata.csv' ) 
gn = MoviesGenra[mgenra] 
A = RECOMMENDER(Movies, [gn,5]) 
opt = A.Model(A.listOfValues) 
recomendedItem = A.outPutHandling(opt) 
return recomendediItem 


##### weather API Function 


def get_temperature(city): 
r = requests.get( 'http://api.openweathermap.org/data/2.5/weather?q='+city+',eg&appid= ') 
json_object = r.json() 
temp_k = float(json_object['main']['temp']) 
temp_c = temp_k - 273.15 
return temp_c 


Figure 4. Movie recommender system and weather API function calls 


3.2. Home appliances control 

With the help of a classification model, it was possible to predict the user’s behavior in turning 
on/off (e.g., light) in the home automation system based on his behavior with the time of day and the room 
light state throughout several months of the year. Classification models provide an overcome and predictions 
for type of suggestion by training on several samples or data created by saving the user’s behaviour on the 
home lights in a dataset. This dataset will be available after certain time and could be improved with dataset 
rising. Naive Bayes classifier have been employed to provide reasonable accuracy based on the Bayes rule 
and secondary probabilities for on and off events. 

For the web interface, the front-end used to adapt data into graphical interface for user to interact 
with and view data via digital interaction using cascading style sheets (CSS), HTML and Java script for the 
front and back end. Numerous keywords were used such as “web of things”, “internet”, “mobile computing”, 
“web” and concatenation of them. Current surveys on the web of things (WoT)/IoT and mobile sensing were 
also studied for related efforts. 

On the hardware side, Arduino and Raspberry Pi were used. On the Arduino side, we can utilize 
Arduino to control nearly any home appliance and make things simple to use by leveraging NLP to give 
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voice commands to the mobile device or web interface, as previously described. When you’re not at home, 
you can use the internet to control your house’s applications. You may use Arduino to control light, fans, 
air conditioning, doors, and curtains, as well as calculate water flow, electricity usage, and control television 
by setting favorite channels and turning it on when you need to use any functions, or hardware such as 
sensors, motors, and wires. The firmware used was Arduino integrated development environment (IDE), 
and an Arduino uno kit with 32 kB of memory and 16 MHz clocks was utilized as a control signal to switch 
functions on/off through a set of relays. We connect Arduino to the internet using the ethernet shield, which 
is backwards compatible with earlier versions but requires the newer W5500 chip, as well as the ethernet and 
SD libraries (to read and write SD card). 


# Main API ######H#eeHH 


@app.route('/main', methods=['POST']) 
def analyze_data(): 
if not request.json or not 'message' in request.json: 
abort (400) 


message = request.json['message' ] 


EAR = NLP() 
Mou = Mouth() 


EAR.execute(message) 
try: 
if EAR.information['Type'] == 'movie': 
genra = EAR.information[ 'Category'] 
movie = recmmend(genra) 
return jsonify({'message': 'Here is your Movie : '+movie}), 200 
except (RuntimeError, TypeError, NameError , KeyError): 


pass 
# ### Light Cycle 
try: 
if EAR.information['Appliance'] == 'light' and EAR.information['State'] == 'on': 
code = lightCode0N[EAR.information['Location']] 
print(code) 


send.Conect(clientName) 
send.send(clientName , TOPIC , code) 
send.disconnect(clientName) 
elif EAR.information['Appliance'] == 'light' and EAR.information['State'] == 'off': 
code = lightCodeOff[EAR.information[ 'Location']] 


send.Conect(clientName) 
send.send(clientName , TOPIC , code) 
send.disconnect(clientName) 
except (RuntimeError, TypeError, NameError , KeyError): 
pass 


Figure. 5. Main API function calls 


In this paper, MQTT is used to connect Raspberry Pi IoT devices and control them from anywhere in 
the world. MQTT is a lightweight messaging protocol that is ideal for communication between connected 
devices in the internet of things. MQTT is made up of three components: a broker, a publisher, and a subscriber. 
A broker is a type of intermediary that facilitates communication between devices. A publisher is a device 
that sends messages. A subscriber listens to the messages sent by the publisher. There is one more important 
component in MQTT known as topic. A topic is necessary for communication between different devices. 
For example, “device a” wants to send a message to “device b”. To do so, there must be a point of 
convergence between the two, which is the topic. Cloud MQTT is a broker service that provides free MQTT 
communication for a certain number of devices. Here, the system is linked to a smart mirror control that 
displays the time, date, weather outdoors, and impending calendar appointments. The hardware requirements 
integrated with the Raspberry Pi 3 are liquid crystal display (LCD) display, wooden frame, high definition 
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multimedia interface (HDMI) and Power supply programmed with python integrated development and 

learning environment (IDLE) 2. Many libraries were used to construct the smart mirror program: 

a) Tkinter library: to create graphical user interface (GUI) programs. 

b) Locale library: open access to the portable operating system interface (POSIX) (to access to locale 
database). 

c) Threading library: accelerate the program by doing many parallel operations instead of doing operations 
sequentially. 

d) Time library: to connect the time to the date of the personal computer (PC). 

e) Requests library: to open and retrieve links. 

f) Java script object notation (JSON) library: to send links. 

g) Traceback library: (most recent call last) print exception tracebacks inside your programs. 

h) Feedparser library: to analyze the abstracts (e.g. rich site summary (RSS)). 

i) Python imaging library (PIL) library: to handle images in terms of image processing in an easy way. 

j) | Context manager: to run two processes with one another, if in a set of instructions among them Google 
News, free geoip were used in smart mirror for getting news to show (my internet protocol (IP), 
latitude, longitude, country code, country name) followed by dark sky to take the weather (create 
account on it then taking the link which on it my weather). 

Smart home applications can be accessed by mobile application or web application, with user 
authentication performed by collecting the user’s face image and comparing it to the server database. 
After that, the server corresponds the result of image processing algorithms via computer vision module 
using OpenCV library to identify the user (or user can authorize and access by entering his own username 
and password). The authentication system is a one-time process that increases the credibility of the system. 
The user voice commands are converted from speech to text and then sent via an HTTP request to our server. 

At the user login case, if user already had an account. In the event that the user needs to log in, 
the system generates a login form and asks the user to enter his face photo, after which the system validates 
the authorization. If the user is authorized, the system will open and the user will be successfully logged in; 
if the user is not authorized, an error message will be displayed and the login will be ended at the not 
recognized face. For clarification login flowchart is diplayed in Figure 6. 


Request Login 


Response chatting room 


Web site 


Check 
login 
device 


Username 


Scan user 
& Password 


face 


Un Authorized 


Check 
author 


Authorized 


Open system (Application) 


Figure 6. The login flowchart 
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In the recommender use case, when the user wants to ask for item recommendation, the user first 
makes a service request. The system then provides a chat form in which the user requests a recommendation. 
After that, the system analyzes and understands the item and recommends that it be displayed if it is available 
on the system; if it is not available on the system, an error message is displayed. 

In the get weather use case, when the user wants to ask for the weather, the system then performs 
analysis and understanding of the user’s request before checking the weather condition. Finally, the weather is 
displayed if the weather condition is available. If not available, a system error message return. The appliance 
control use case is when the user wants to ask for application control (e.g., light, fans, air-condition, doors, 
curtains, water flow calculate, calculate electricity consumption, and control TV). First, the system generates 
the chat form, user asks for the application control. After that, system performs application control and 
switch application on/off. If its available, system displays the item. If not, a system error message return. 


4. CONCLUSION 

To operate home appliances, the majority of present home automation systems follow a series of 
procedures or commands. To overcome the constraints of previous automation systems (such as the necessity to 
connect the system to the internet and construct an AI recommendation system), web and android applications 
based on voice commands have been utilized to make controlling household appliances more flexible by 
interpreting these commands into reality using Arduino and Raspberry Pi boards. In addition, a recommendation 
system has been developed to predict user’s behavior based on the data saved in the dataset. Future works 
include addiding more home appliances with with approparite ML and AI algorithms. 
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